{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TF-IDF demo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h2o\n",
    "\n",
    "h2o.init()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data sources:\n",
    "\n",
    "* https://github.com/h2oai/h2o-3\n",
    "* https://en.wikipedia.org/wiki/Ice_hockey\n",
    "* https://en.wikipedia.org/wiki/Antibody"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Document                                                                                                                                                                                                                                           </th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>H2O is an in-memory platform for distributed, scalable machine learning. H2O uses familiar interfaces like R, Python, Scala, Java, JSON and the Flow notebook/web interface, and works seamlessly with big data technologies like Hadoop and Spark.</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>Ice hockey is a contact team sport played on ice, usually in a rink, in which two teams of skaters use their sticks to shoot a vulcanized rubber puck into their opponent's net to score goals. The sport is known to be fast-paced and physical.  </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>An antibody (Ab), also known as an immunoglobulin (Ig), is a large, Y-shaped protein produced mainly by plasma cells that is used by the immune system to neutralize pathogens such as pathogenic bacteria and viruses.                            </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import OrderedDict\n",
    "\n",
    "documents = [\n",
    "    'H2O is an in-memory platform for distributed, scalable machine learning. H2O uses familiar interfaces like R, Python, Scala, Java, JSON and the Flow notebook/web interface, and works seamlessly with big data technologies like Hadoop and Spark.',\n",
    "    'Ice hockey is a contact team sport played on ice, usually in a rink, in which two teams of skaters use their sticks to shoot a vulcanized rubber puck into their opponent\\'s net to score goals. The sport is known to be fast-paced and physical.',\n",
    "    'An antibody (Ab), also known as an immunoglobulin (Ig), is a large, Y-shaped protein produced mainly by plasma cells that is used by the immune system to neutralize pathogens such as pathogenic bacteria and viruses.'\n",
    "]\n",
    "doc_ids = list(range(len(documents)))\n",
    "\n",
    "input_frame = h2o.H2OFrame(OrderedDict([('DocID', doc_ids), ('Document', documents)]),\n",
    "                            column_types=['numeric', 'string'])\n",
    "input_frame.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TF-IDF with pre-processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word                                                                                                                                                                                                                                               </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>an antibody (ab), also known as an immunoglobulin (ig), is a large, y-shaped protein produced mainly by plasma cells that is used by the immune system to neutralize pathogens such as pathogenic bacteria and viruses.                            </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>h2o is an in-memory platform for distributed, scalable machine learning. h2o uses familiar interfaces like r, python, scala, java, json and the flow notebook/web interface, and works seamlessly with big data technologies like hadoop and spark.</td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>ice hockey is a contact team sport played on ice, usually in a rink, in which two teams of skaters use their sticks to shoot a vulcanized rubber puck into their opponent's net to score goals. the sport is known to be fast-paced and physical.  </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from h2o.information_retrieval.tf_idf import tf_idf\n",
    "\n",
    "tf_idf_out = tf_idf(input_frame, \"DocID\", \"Document\", False, False)\n",
    "tf_idf_out.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import DisplayObject, display\n",
    "VALUES_CNT_TO_SHOW = 3\n",
    "\n",
    "def tf_idf_output_summary(tf_idf_out):\n",
    "    for doc_id in doc_ids:\n",
    "        sorted_doc_tf_idfs = tf_idf_out[tf_idf_out['DocID'] == doc_id].sort(by='TF-IDF')\n",
    "        print('The highest TF-IDF values for document ' + str(doc_id) +':')\n",
    "        display(sorted_doc_tf_idfs.tail(VALUES_CNT_TO_SHOW))\n",
    "        print('The lowest TF-IDF values for document ' + str(doc_id) +':')\n",
    "        display(sorted_doc_tf_idfs.head(VALUES_CNT_TO_SHOW))\n",
    "        print('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The highest TF-IDF values for document 0:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word                                                                                                                                                                                                                                               </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>h2o is an in-memory platform for distributed, scalable machine learning. h2o uses familiar interfaces like r, python, scala, java, json and the flow notebook/web interface, and works seamlessly with big data technologies like hadoop and spark.</td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The lowest TF-IDF values for document 0:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word                                                                                                                                                                                                                                               </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>h2o is an in-memory platform for distributed, scalable machine learning. h2o uses familiar interfaces like r, python, scala, java, json and the flow notebook/web interface, and works seamlessly with big data technologies like hadoop and spark.</td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "The highest TF-IDF values for document 1:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word                                                                                                                                                                                                                                             </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>ice hockey is a contact team sport played on ice, usually in a rink, in which two teams of skaters use their sticks to shoot a vulcanized rubber puck into their opponent's net to score goals. the sport is known to be fast-paced and physical.</td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The lowest TF-IDF values for document 1:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word                                                                                                                                                                                                                                             </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>ice hockey is a contact team sport played on ice, usually in a rink, in which two teams of skaters use their sticks to shoot a vulcanized rubber puck into their opponent's net to score goals. the sport is known to be fast-paced and physical.</td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "The highest TF-IDF values for document 2:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word                                                                                                                                                                                                                   </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>an antibody (ab), also known as an immunoglobulin (ig), is a large, y-shaped protein produced mainly by plasma cells that is used by the immune system to neutralize pathogens such as pathogenic bacteria and viruses.</td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The lowest TF-IDF values for document 2:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word                                                                                                                                                                                                                   </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>an antibody (ab), also known as an immunoglobulin (ig), is a large, y-shaped protein produced mainly by plasma cells that is used by the immune system to neutralize pathogens such as pathogenic bacteria and viruses.</td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tf_idf_output_summary(tf_idf_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TF-IDF without pre-processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Document    </th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>H2O         </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>is          </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>an          </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>in-memory   </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>platform    </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>for         </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>distributed,</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>scalable    </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>machine     </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>learning.   </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessed_data = [(doc_id, word) for doc_id, document in enumerate(documents) for word in document.split()]\n",
    "\n",
    "preprocessed_input_frame = h2o.H2OFrame(preprocessed_data,\n",
    "                                        column_names=['DocID', 'Document'],\n",
    "                                        column_types=['numeric', 'string'])\n",
    "preprocessed_input_frame.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word   </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>(Ab),  </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>(Ig),  </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>An     </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>Flow   </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>H2O    </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">1.38629 </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>Hadoop </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>Ice    </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>JSON   </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>Java,  </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>Python,</td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_idf_out = tf_idf(preprocessed_input_frame, 'DocID', 'Document', preprocess=False)\n",
    "tf_idf_out.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The highest TF-IDF values for document 0:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word  </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>works </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>H2O   </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">1.38629 </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>like  </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">1.38629 </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The lowest TF-IDF values for document 0:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word  </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>and   </td><td style=\"text-align: right;\">   3</td><td style=\"text-align: right;\">0       </td><td style=\"text-align: right;\">0       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>is    </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0       </td><td style=\"text-align: right;\">0       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>an    </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.287682</td><td style=\"text-align: right;\">0.287682</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "The highest TF-IDF values for document 1:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word  </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>in    </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\"> 1.38629</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>sport </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\"> 1.38629</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>their </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\"> 1.38629</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The lowest TF-IDF values for document 1:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word  </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>and   </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0       </td><td style=\"text-align: right;\">0       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>is    </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0       </td><td style=\"text-align: right;\">0       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>known </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.287682</td><td style=\"text-align: right;\">0.287682</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "The highest TF-IDF values for document 2:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word    </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>viruses.</td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>as      </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">1.38629 </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>by      </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">1.38629 </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The lowest TF-IDF values for document 2:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word  </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>and   </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0       </td><td style=\"text-align: right;\">0       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>is    </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0       </td><td style=\"text-align: right;\">0       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>a     </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.287682</td><td style=\"text-align: right;\">0.287682</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tf_idf_output_summary(tf_idf_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case insensitive TF-IDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "input_frame = h2o.H2OFrame(OrderedDict([('DocID', doc_ids), ('Document', documents)]),\n",
    "                            column_types=['numeric', 'string'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word  </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>(ab), </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>(ig), </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>a     </td><td style=\"text-align: right;\">   3</td><td style=\"text-align: right;\">0.287682</td><td style=\"text-align: right;\">0.863046</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>a     </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.287682</td><td style=\"text-align: right;\">0.287682</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>also  </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>an    </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.287682</td><td style=\"text-align: right;\">0.287682</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>an    </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.287682</td><td style=\"text-align: right;\">0.575364</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>and   </td><td style=\"text-align: right;\">   3</td><td style=\"text-align: right;\">0       </td><td style=\"text-align: right;\">0       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>and   </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0       </td><td style=\"text-align: right;\">0       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>and   </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0       </td><td style=\"text-align: right;\">0       </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_idf_out = tf_idf(input_frame, 'DocID', 'Document', case_sensitive=False)\n",
    "tf_idf_out.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The highest TF-IDF values for document 0:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word  </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>works </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>h2o   </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">1.38629 </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>like  </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">1.38629 </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The lowest TF-IDF values for document 0:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word  </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">  IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>and   </td><td style=\"text-align: right;\">   3</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">       0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>is    </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">       0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      0</td><td>the   </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">       0</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "The highest TF-IDF values for document 1:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word  </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>in    </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\"> 1.38629</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>sport </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\"> 1.38629</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>their </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\"> 1.38629</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The lowest TF-IDF values for document 1:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word  </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">  IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>and   </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">       0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>is    </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">       0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td>the   </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">       0</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "The highest TF-IDF values for document 2:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word    </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">     IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>y-shaped</td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">0.693147</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>as      </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">1.38629 </td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>by      </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">0.693147</td><td style=\"text-align: right;\">1.38629 </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The lowest TF-IDF values for document 2:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  DocID</th><th>Word  </th><th style=\"text-align: right;\">  TF</th><th style=\"text-align: right;\">  IDF</th><th style=\"text-align: right;\">  TF-IDF</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>and   </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">       0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>is    </td><td style=\"text-align: right;\">   2</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">       0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">      2</td><td>the   </td><td style=\"text-align: right;\">   1</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">       0</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tf_idf_output_summary(tf_idf_out)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3rc1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
